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ABSTRACT 


Augmented Reality (AR) technology and games can enhance motivation for learning. When combined with mobile 
devices, AR technology can promote authentic learning in outdoor environments, such as urban parks. These spaces can 
be used for Science Education, particularly, for environmental education and nature conservation. The EduPARK project 
combines these elements and follows a design-based research approach to develop an interactive mobile AR game to be 
explored by students, teachers and the general public, as visitors, in a specific urban park, integrating four 
interdisciplinary educational guides. The app development involved four cycles of user testing and evaluation for 
progressive refinement, according to the users’ feedback in each cycle. The focus of this paper is to analyze the users’ 
perceptions in the two last cycles of the app refinement, regarding its learning value and usability. The users were 
students of different school levels (24 of Basic Education; and 46 of Higher Education). Data collection involved a focus 
group interview, a questionnaire and the app’s usage data. Content analysis, descriptive statistics, and System Usability 
Scale (SUS) computing were conducted. Results revealed that the EduPARK app promotes learning, enjoyment and is 
easy to use. It achieved an excellent usability, according to younger students (85.8 of average SUS) and a good usability 
according to older students (70.9). From the last refinement cycle, the app’s final version has emerged, which is freely 
available to the public in the Google Store. In the future, more evaluation experiences are needed to better understand the 
benefits of this mobile AR game for learning in urban parks. 
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1. INTRODUCTION 


In a technology driven society, educators can take advantage of the pervasiveness of technological devices to 
innovate their educational practices and, therefore, to promote authentic learning. For instance, mobile 
devices, such as smartphones, can be used as supporting tools of learning activities and can open up new 
opportunities for digital learning in formal, informal (Parsons 2014) and non-formal settings. The use of 
mobile games in formal education has been shown to promote student engagement for deeper and authentic 
learning (Huizenga et al. 2009). When combined with emerging technologies, such as Augmented Reality 
(AR), mobile devices are claimed to also create conditions to promote students' engagement with learning 
(Giannakas et al. 2017). Additionally, the recent proliferation of mobile devices and applications (app) makes 
AR technology accessible to support learning anytime anywhere, even in outdoor environments, such as 
urban parks. 

The EduPARK project, ‘Mobile Learning, Augmented Reality and Geocaching in Science Education — an 
innovator design-based research project’ (http://edupark.web.ua.pt), emerged in the above-mentioned 
context, where original technologies can be used to promote education. The project’s aim is to create 
innovative practices to promote interdisciplinary learning in Science Education, articulated with the school 
curriculum. For that, Geocaching-based principles are integrated in the game-like activities in a specific 
urban park, the Infante D. Pedro Park in Aveiro (Portugal), which are supported by mobile AR technologies. 
The project developed an interactive AR mobile app that integrates four educational guides — two for 
different levels in Basic Education, one for Secondary and Higher Education, and another for the general 
public. The latter is also available in English, so that foreign people (external visitors) can explore the app. 
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Each guide comprises a game with interdisciplinary questions and educational challenges that prompt for the 
search of interesting places in the Park, so the users can learn while enjoy a healthy walk along the Park. The 
City Council collaboration allowed the installation of 32 plant identification plaques in the Park with AR 
information on biological & curiosity aspects of plant species, historical references and a map (Pombo, 
Marques, Loureiro, et al. 2017). The app can, thus, be used autonomously, using the game mode or exploring 
it freely, promoting authentic learning. 

The project has been organizing and implementing several activities in the Park, frequently integrated in 
outreach events of its host institution, the University of Aveiro, to test, evaluate and improve the app and its 
included educational guides, following a design-based research approach (Parker 2011) with four cycles of 
refinement. The app’s description and the first two cycles of development and evaluation were described 
before (Pombo et al. n.d.; Pombo, Marques, Carlos, et al. 2017; Pombo & Marques 2017). The main purpose 
of this paper is to evaluate the 3“ and 4" cycles of the app, regarding the users’ perspectives on the 
educational value for authentic learning, as well as the app’s usability. 

The paper is structured in the following sections: (i) literature review on mobile AR and its relevance in 
education contexts; (ii) research approach, explaining the 3™ and 4" cycles of the design-based methodology; 
(iii) presentation and discussion of main results; and (iv) final remarks and recommendations for future work. 


2. THEORETICAL FRAMEWORK 


According to Crompton (2013), mobile learning denotes a way of learning across contexts, through social 
and content interactions, with the support of mobile devices. Among their affordances are the mobility 
(Parsons 2014); the instant access to learning content (Giannakas et al. 2017); and the hardware and 
applications’ panoply that support orientation, measuring, registering, organizing and communicating, among 
other activities, enhancing contextual and situated learning (Parsons 2014). Regarding its pitfalls, the 
literature mentions it entrenches digital divides regarding technology access, technological skills and learning 
competencies (Parsons 2014). 

Additionally, AR technology supported by mobile devices creates learning opportunities that go beyond 
physical environments (Parsons 2014). AR allows overlying virtual objects in a real world environment in 
real time, producing a new experience. There are already a number of developed AR games for mobile 
devices that support authentic learning in outdoor settings, such as the ones briefly described below. A trend 
seems to be the use of AR apps with some sort of geo-location mechanisms. 

Alien contact!, developed by Dunleavy et al. (2009), is a curriculum relevant and narrative-driven, 
inquiry-based AR simulation. In this game, students move around a physical location to get closer to digital 
artefacts displayed in a map, triggering video, audio, and text files. To successfully discover why the aliens 
had landed and solve academic challenges (subtasks involving math, language, arts, and scientific literacy 
skills), students in the same group play different roles and share information. The results included high 
engagement and motivation supported by the novelty of the use of handheld computers and GPS to learn; 
collection of data outside the classroom (more authentic learning environment); development of physical 
space orientation skills; and distributed knowledge, positive interdependence and different roles. The main 
difficulties pointed out were the GPS errors; screen visualization and audio listening in the outdoors; the high 
management requirements for teachers (to maintain the activity flowing); student cognitive overload and 
strong competition between teams. 

The AR butterfly ecological learning system (Tarng & Ou 2012), developed using AR and mobile 
technologies, follows a game-based and mobile learning approach to teach students about butterfly species, 
ecology and conservation, in a campus environment. The AR was triggered by GPS coordinates 
(location-based AR), so students could observe virtual butterflies around nectar plants when they approach 
certain locations. The application also allows the breeding of virtual butterflies to allow observation of the 
butterfly life cycle and support understanding of their growing process. Students can autonomously use this 
application. 

The EcoMOBILE (Kamarainen et al. 2013) is another project that combines mobile platforms and AR to 
provide a more interactive way to learn in outdoor contexts and increase student motivation and engagement. 
In EcoMOBILE the information is accessed by GPS triggers. The information ranges from interactive media, 
such as images, videos, 3D models and collaborative quizzes (multiple choice and open-ended questions) to 
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AR visualizations. The authors claim that the use of these technologies keeps the students motivated and 
allows them to explore the field at their own pace, freeing the teachers to act as a facilitator and move around 
to check the progress of the different groups. Although these experiences have a lot of positive aspects, an 
issue found was that some groups just speeded through the activity without reading or fully understanding the 
accessed contents. 

The ZooEduGuide (Srisuphab et al., 2014) is a mobile application to enhance the zoo visits experience in 
Thailand. This app provides an interactive way to explore a zoo by combining mobile devices and AR. It has 
three main modes: (i) learning, which presents information about the animals via rich interactive multimedia 
content, where the information is accessed by scanning QR codes; (ii) educational games, animal sound 
identification and trivia quiz, where the games’ scores are ranked adding a social and competitive 
component; and (ili) the visit zoo mode, which is a zoo guide where visitors can visualize the map and 
customize it with their own points of interest and visit path, and check the event schedule. The application 
makes use of AR to guide the visitor inside the zoo by showing points of interest in the camera view. 

The AR competitive game developed by Hwang et al. (2015) is based on the traditional board game 
concept. The players need to roll a digital dice, but in this game they move around in a butterfly garden. At 
each location, students need to answer questions or complete a mini-game. Hwang et al. (2015) recommend a 
three-step design procedure for applying a competitive gaming approach to AR-based outdoor activities: (1) 
Select the activities that require students to explore or make observations in real-world contexts; (2) Prepare a 
set of questions related to the real-world contexts for the competitive game; and (3) Determine the location 
and content for each AR-based events. 

Overall, the above-mentioned initiatives provide important insights into the use of AR mobile technology 
in outdoor educational settings. For example, AR is recognized as a technology that might enhance student 
interest and motivation, as well as promote self-learning. AR can support the understanding of complex and 
abstract concepts and, when combined with game-based learning, students may be more willing to overcome 
challenges and learning difficulties. 


3. METHODOLOGICAL OPTIONS 


The EduPARK project follows a qualitative interpretative methodology (Amado 2014) with multiple 
iterations for refinement and evolution of a prototype, justifying a design-based research approach (Parker 
2011), as shown in Figure 1. The prototype is a game-like app for mobile devices with AR and the main aim 
is to collect students’ perceptions regarding the app’s learning value and usability. 
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Figure 1. The EduPARK Project Methodology: Design-Based Research (Parker 2011), Comprising four Refinement 
Cycles 


25 


ISBN: 978-989-8533-76-0 © 2018 


3.1 Context of Development 


The first two cycles of prototype development and evaluation were described in other works (Pombo et al. 
n.d.; Pombo, Marques, Carlos, et al. 2017; Pombo & Marques 2017), so this work presents only a brief 
description of those cycles below. 

In summary, the initially developed version of the app comprised interdisciplinary quizzes with 
multiple-choice questions and content (text, audio or image) aligned with the curriculum, feedback to the 
users’ answers and scores. The questions included contents from diverse subjects, such as biology 
(particularly about botanics and zoology), mathematics, local history and physical education. Two 
educational guides, one for the 1“ and another for the 3"! Cycle of Basic Education, were developed to be 
played by teams of students in a friendly competition approach. These were prompted to search for and locate 
three physical caches in the Park (Pombo et al. n.d.; Pombo, Marques, Carlos, et al. 2017). During the 
prototype development, a heuristic evaluation was made, based on the ‘usability assessment of interfaces of 
mobile devices’ framework (Neto & Pimentel 2013), which led to improvements prior to tests with users. 

In the 1“ refinement cycle, under the 2016 UA Open Week of Science and Technology, 74 students (aged 
9-10 and 13-14) played with the prototype of the EduPARK app game in the Park and provided feedback. 
Students reported a positive perception regarding the app’s usability and value for authentic learning. For 
example, a 4" school level student reported: ‘I like [this app] because we can learn more about the plants and 
Nature.’ One of the accompanying teachers, while speaking with her students at the end of the activity, 
mentioned: ‘With this game you learn several subjects. You have Mathematics, Sciences, Botany... And you 
apply, in the outdoors, the knowledge you learnt in the classroom.’ However, some inconsistencies were 
pointed out, such as GPS precision errors, which are widely reported in other studies (Akgayir & Akgayir 
2017), and difficulties in the use of some AR markers. Also, the students suggested to include animations, 
audios and videos (Pombo et al. n.d.; Pombo, Marques, Carlos, et al. 2017), supporting Srisuphab et al. 
(2014), who claim that information in AR mobile apps should be available in different formats. 

The 2" cycle involved the refinement of the 1“ version of the app, namely the development of more 
reliable markers. The new markers served as a way of triggering AR content, instead of being used as a 
checkpoint of the game, as in the 1* cycle. Also, physical plaques were developed comprising two main 
functions: i) botanical species identification; ii) markers to trigger AR contents. This option allows 
addressing difficulties in the use of the initial markers. Additionally, markerless tracking was also 
implemented, using 2D imagery, as historical tiles, already located at the Park, increasing the number of 
opportunities of authentic learning. Under this cycle, two future primary teachers, enrolled in a Master’s 
degree in Basic Education, developed an educational guide for young pupils (aged 9-10). This guide was 
integrated in the new version of the EduPARK app and was tested in two events: i) the UA Open Campus 
2017, when 23 students (aged 15 to 19) tested the app in the University campus to point strengths and 
weaknesses; and ii) an activity in the Park involving the 1“ Cycle class with 21 pupils (aged 9-10) assigned to 
the two future primary teachers (masters’ students) mentioned above. 

Once more, the students reported a positive perception of the app usability and value for learning: ‘I 
would like to play the game again because it was really fun to use cell phone in a lesson to review the content 
we’ve learnt’ (4"" school level student). Younger students proposed improvement suggestions exclusively 
regarding the game features, such as to include riddles to unlock treasure chests, to include time limit for 
question answering or to add more questions. Older students also pointed some usability-related suggestions, 
like to increase the app’s responsiveness to the touch. 


3.2 The 3" Cycle 


Among the new features introduced in this app version was the inclusion of audios, videos and animations, to 
diversify the formats of information, as suggested by pupils in the 1“ refinement cycle. Other improvement 
was related to the GPS-based search for virtual caches (treasure chests) containing virtual objects and 
prompted by riddles. Users received instructions to search for a virtual cache and received GPS-based 
information about their distance to it. A new educational guide with an interdisciplinary quiz was developed 
for the 2" cycle of Basic Education. The students’ suggestion of limiting the time for question answering, 
although it could increase the game’s competitiveness, it was not considered in the revised app, as the 
literature has long shown that time to think is relevant for learning performance (Tobin & Capie 1983). 
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Students with 10 to 12 years-old tested and evaluated this app version in the Park under the UA Summer 
Academy 2017, organised after the end of the school year, in a non-formal context of learning. The 
integration of the app’s test in this event was as a way to gather a convenience sample of users, as students 
were enrolled in this event. The sample was composed by 16 girls and 8 boys (total of 24), who had just 
finished the 5" year (5 students) and the 6" year of Basic Education (19 students), in eight schools of the 
region. The students tested the app in the project’s mobile devices, in 8 groups of 3 elements, each one 
accompained by an accompanying researcher. The test lasted 60 to 90 minutes, approximately. 


3.3 The 4" Cycle 


In the last refinement cycle included the guide for the Secondary and Higher Education according to Science 
curricular contents related to those levels. The search for caches was based in riddles to find a specific 
marker, instead of using GPS to avoid problems of geolocalization. The change in the AR triggering 
technology is due to the frequently reported GPS precision errors, both in the literature (Akgayir & Akgayir 
2017) and in previous app refinement cycles. Addittionally, 32 permanent plaques of botanic specimens 
identification, with AR markers, were installed in the Park (with proper authorization of the City Council) to 
allow autonomous exploration of the app by users, as proposed in Pombo and Marques (2017). 

Under the curricular unit ‘Nature Integrated Sciences’, the app was tested in the Park by 46 Basic 
Education graduating students, future-teachers, of which 45 were female students. Ages varied from 18 to 42 
years-old, with an average age of 21 years. The students’ academic backgrounds varied, being 25 from 
humanities studies, 12 from sciences, and the remaining from other areas, such as Arts or Sports. For the first 
time, the app was tested with the users’ own mobile devices (instead of the project’s smartphones), which 
involved a wide typology of devices and created some technical problems. Students tested the app in 16 
groups of 3/4 elements during 74 to 110 minutes, approximately. 

In both the 3™ and 4" refinement cycles, data were collected through a focus group interview (FGI), a 
questionnaire and the app’s usage data. Both, FCI and questionnaires, were described before (Pombo et al. 
n.d.; Pombo, Marques, Carlos, et al. 2017) and were applied immediately after testing. Data analysis 
involved: 1) content analysis of the FGI; and ii) descriptive statistics of the app’s usage data and of the 
questionnaire’s answers. Data were triangulated to provide a comprehensive knowledge of the students’ 
perceptions regarding the app and their learning. This analysis will be presented in the next section. 


4. RESULTS AND DISCUSSION 


In the 3" refinement cycle, the collected data allowed profiling the users (24 students enrolled in the UA 
Summer Academy 2017) regarding their demographics and mobile devices proficiency. It is possible to 
observe that 23 students claimed to own a personal mobile phone, from which 19 were smartphones. Most of 
them use the device for about 30 and 59 minutes (9 students) or 15 and 29 minutes (7) per day. Besides 
phone calling and text messaging (both selected by 21), the students use it for: gaming (19 students), video 
watching (15), internet searching (14), listening to music (12), and using social networks (11). Hence, these 
results point that the majority of the students was skilled with mobile devices. These results seem to support 
the literature, regarding the proliferation of mobile devices (Johnson et al. 2014), especially in what concerns 
young population. Besides that, young students reported feeling enthusiasm and enjoyment by using the app. 
For example, one student commented ‘I really enjoyed playing this game and I would like to do it again’ 
(questionnaire student’s cote) and another stated that it ‘gave us knowledge, but at the same time we also 
could have fun.’ (FGI student’s cote). Additionally, some students also noted they would like the game to last 
longer, despite the groups’ average game time being of about 76 minutes. Such results are in line with the 
literature that claims that student engagement and motivation to learn can be promoted by the use of games 
(Freitas, 2008), supported by mobile (Kamarainen et al., 2013) and AR technologies (Akgayir & Akcayir 
2017; Dunleavy et al. 2009). 

The students’ perception of the app’s usability was positive; and 22 students agreed or strongly agreed 
with the statement ‘I thought the app was easy to use’ and 18 disagreed or strongly disagreed with the 
statement ‘I found the app more difficult to use than it should be.’ The SUS score was computed individually 
and, afterwards, an average value was computed for all the questionnaires. Values ranged from 60 to 100, 
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with an average of 85.8. Therefore, according to the classification of Bangor et al. (2009), this version of 
EduPARK app achieved an excellent usability. 

Concerning the technology-related difficulties, students reported: 1) GPS signal malfunctions — ‘The 
distance meters [from the virtual cache] sometimes didn’t work very well’ (FGI student’s cote); 1i) 
displeasure in having to keep the mobile device camera pointed at the marker during the exploration of the 
AR contents — ‘I liked it, but (...) when we saw the markers, we had to keep pointing [the mobile phone] at 
them. It would be nicer to keep the information [in the screen]’ (FGI student’s cote); and iii) a few difficulties 
in handling the app — ‘When we were searching for caches, it would appear the “quit” button, which was 
similar to the “continue” button, and we would press it by mistake’ (FGI student’s cote). 

In what concerns the game features, students referred they would like: i) to see the collected virtual 
objects having a goal — ‘I think that the objects inside the caches could have some kind of use’ (FGI student’s 
cote); and ii) to include different paths in the game — ‘To have different routes in the app’ (FGI student’s 
cote). 

Regarding learning obstacles, students reported some difficulties with the vocabulary used in the app — 
‘we didn’t know several words’ (FGI student’s cote) and that they found some quiz questions challenging — 
‘in some questions we had to think harder.’ (FGI student’s cote). On the other hand, one of the advantages 
acknowledged by the students was that they were more motivated to learn in outdoor settings — ‘It is cooler in 
the outdoors’ (FGI student’s cote) — and that the game increased motivation due to the authenticity of the 
learning environment — “This way we are more motivated to do the calculus [of an object’s area]. We know it 
will have a purpose.’ (FGI another student’s cote). Another relevant benefit was learning more about the 
local Park — ‘For example, I’ve never seen that fountain with fish, where the cache was hidden’ (FGI 
student’s cote). More specifically, regarding the Mathematics’ learning, student claimed this activity 
‘captivated us to Math’ (FGI student’s cote), despite being in school holidays period. Figure 2 shows the 
students’ perspectives regarding the Mathematics questions in the app, which are strongly positive. 
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Figure 2. Third Refinement Cycle: Students’ Opinion Regarding the Learning Value of the app 


In the last refinement cycle, the 46 graduation students/future-teachers reported to frequently 
(37 students) or occasionally (6) use mobile devices to promote their own learning. They also stated to use 
mobile devices in the University (22) and their own home (22) for learning purposes, considering that mobile 
devices allow an easy and quick way of accessing information; although they acknowledged the increase of 
distracting factors and the risk of becoming addicted to this mobile technology. 

Regarding the EduPARK app, students reported positive perceptions. For example, its learning value was 
acknowledged by 35 students, as illustrated here: ‘The EduPARK app is advantageous, essentially, from the 
pedagogic point of view.’ (questionnaire student’s cote) and ‘[with this app] we can see that in each [Park] 
corner there is something to learn and, thus, enrich our knowledge.’ (questionnaire student’s cote). 
Additionally, they also referred the app’s value in what respect intrinsic motivation (32 students), 
engagement (35), authentic learning (33), lifelong learning (28), and conservation habits (31). The average 
game time was 88 minutes, for a park path with 35 quiz questions, and still there were students suggesting to 
add more questions to the game. 

The students’ perception of the app’s usability was positive; and 31 students agreed or strongly agreed 
with the statement ‘I thought the app was easy to use’ and 35 disagreed or strongly disagreed with the 
statement ‘I found the app more difficult to use than it should be’. The SUS scores ranged from 55 to 95, 
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with an average of 70.9, achieving a good usability, lower than in the previous cycle. This may be explained 
by two factors: i) the users were older (future educational professionals) and maybe more critical than 
younger children, which is not a new result, as the literature has had found that SUS scores tend to decrease 
with increasing age (Bangor, Kortum, & Miller, 2009); and 11) the diversity of cell phones used in this test 
turn it more challenging, from the technological point of view. 

Concerning the technology-related difficulties, students reported: i) the visualization of images and 
videos, due to the heavy storage space it requires from the device; ii) some lack of responsiveness of the app; 
iii) the high battery energy requirements; and iv) some app tools did not work (such as the app’s compass). 
Graduating students also suggested to include more AR contents and to have the app available for IOS 
systems. 


5. FINAL REMARKS 


The EduPARK project has been developing an interactive mobile AR game to promote authentic learning in 
a specific Park. The main relevance of the EduPARK project is the articulation between research and 
development, professional practices, and initial and advanced training, as well as its innovation concerning 
outdoor learning strategies, in formal, informal, and non-formal contexts. This research also prompts to 
capitalize urban spaces’ educational value, in association with the local City Council. 

This work reports the 3" and 4" refinement cycles of a design-based research for this app development, 
with the aim of accessing the users’ perspectives of the learning value and usability of this app. Hence, it was 
tested by groups of 10 to 12 aged students, who were enrolled in the UA Summer Academy 2017 and by 
Higher Education students, future-teachers, from the Basic Education Graduation Course of the UA. The 
results show that students enjoyed and were enthusiastic about the app. However, some difficulties with the 
technology were reported in the Summer Academy experience, such as some cases of GPS malfunction, 
limiting some tasks prompted by the app (the search for the virtual caches) and in handling some functions of 
it. These difficulties were taken in consideration in the following app refinement. Concerning learning, 
students referred: i) they found the app’s quiz challenging in a positive way, e.g., it motivated them to learn; 
and 11) they could learn in a situated way. 

The EduPARK app was released to the public and it is freely available in the Google Store 
(http://edupark.web.ua.pt/app). In spite of the positive results so far, regarding the users’ perspectives about 
the learning value and usability of this mobile AR game, the samples were convenience ones and the number 
of users in each target public was scarce. In future the project intends to organize more activities in order to 
collect systematic data that might be used to better understand the benefits of using AR mobile technology in 
outdoor settings for learning and for lifelong learning. Additionally, the team expects to triangulate data from 
different target public besides students, such as teachers, accompanying researchers, external consultants, 
park visitors, and the general public, as the EduPARK app intends to be a resource with impact in schools, 
local community and also in the tourism sector. 

In summary, the EduPARK app will continue to work to achieve its main purpose: to promote an active 
participation of students and the wider community in the construction of their knowledge. Thus, the project is 
fostering authentic learning, both in a formal learning context and in a lifelong learning logic, for the citizens 
in general. Therefore, the EduPARK project is revealing good educational practices in which digital and 
social interactions are valued through the use of innovative technologies that combine, in an articulated way, 
the virtual and real worlds. 
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